Topology-oblivious optimization of MPI broadcast algorithms on extreme-scale platforms

Article history: Available online xxxx Keywords: MPI Broadcast BlueGene Grid'5000 Extreme-scale Communication Hierarchy a b s t r a c t Significant research has been conducted in collective communication operations, in particular in MPI broadcast, on distributed memory platforms. Most of the research efforts aim to optimize the collective operations for particular architectures by taking into account either their topology or platform parameters. In this work we propose a simple but general approach to optimization of the legacy MPI broadcast algorithms, which are widely used in MPICH and Open MPI. The proposed optimization technique is designed to address the challenge of extreme scale of future HPC platforms. It is based on hierarchical transformation of the traditionally flat logical arrangement of communicating processors. Theoretical analysis and experimental results on IBM BlueGene/P and a cluster of the Grid'5000 platform are presented. The Message Passing Interface (MPI) [1] is one of the core building blocks of scientific software libraries for parallel applications. For example, PETSc [2] library, which is used for simulation and modeling in different application domains such as Nanosimulations, Aerodynamics, Geosciences, Computational Fluid Dynamics and others, utilizes MPI for all inter-process data communication operations. Depending on the application, MPI collective communication operations can provide significant performance improvements over MPI point-to-point communication routines. One of the commonly used collective operations, MPI broadcast, is used in a variety of basic scientific kernels such as parallel matrix–matrix multiplication, LU factorization and along with others. During a broadcast operation, the root process sends a message to all other processes in the specified group of processes. The implementations of the broadcast operation in MPICH [3] and Open MPI [4] are typically based on linear, binary, binomial and pipelined algorithms [5]. The linear algorithms are not good for large numbers of processes, the binary and binomial algorithms are not efficient for large data sizes. On the other hand, pipelined algorithms are more efficient for larger numbers of processes and data sizes. Other widely used broadcast algorithms are scatter-ring-allgather and scatter-recursive-doubling-allgather [6], which have been implemented in MPICH.